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ABSTRACT 

The program described here takes as its input a collection 
of lines, vertices and surfaces describing a scene, and 
analyzes the scene into a composition of three-dimensional 
objects. 

The program does not need to know the form {model, or 
pattern) of the objects which are likely to appear* the scene 
is not searched for cubes, wedges or houses, with an a-priori 
knowledge of the form of these objects? rather, the program 
pays attention to configurations of surfaces and lines which 
would make plausible three-dimensional solids f and in this way 
'bodies 1 are identified. 

Partially occluded bodies are handled correctly. The pro- 
gram is restricted to scenes formed by straight lines, where no 
shadows or noise are present, it has been tested in rather 
complicated scenes composed by rather simple objects* Examples 
are given* 
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INTRODUCTION 

We consider visual scenes compoeed by the optical 
imageof ft group of bodies. When s^c!h ;-, SL-ei.e is '^uru'" 
by ft computer through a film spot scanner, image dis- 
sector, or similar dcvioe p it can be treated aa a two-di- 
mensional array of numbers, or as a function of two 

VftliftblM, 

At ft h^hcr level, i\ scene could be meaningfully de- 
scribed as a conglomerate of points, lines and Burf aces, 
with properties (coordinate, slopes, ...) attached to 
them 

Still ft more sophisticated description could use terms 
aoncerriing the bodies or objects which compoee such a 
aoene, indicating their pwtions, inter-relations., etc. 

This pftper describes a program which finds bodies in 
ft scene, presumably formed by three-dimensional ob- 
ject*. Some of them may not be completely visible. The 
picture w presented at a line drawing. 

When SEE — the pretentious name of the program — 
analyse* the scene TRIAL {see Figure 1 'TRIAL'), the 
mult* are: 

(BODY LIS =6:2:1) 
(DODY2,IS:ll:l2;lO) 

(BODY 3- IS ?t S ;S :7 fl £ ;13) 

BEE looks for three-dimcnKonal object* in a two- 
dimensional scene, Thffsccne itself is not obtained from 
ft visual input device, or from an array of intensities or 
brightness. Rather, it is assumed that a preprocessing 
of some sort haa taken place, ftnd the scene to be ana- 
lysed is available In a aymbotie format (to be described 
in a later Section), in terms of potato (vertices), lines 
(edges) | and surface (regions). 

SEE does not: have a pre-concleved idea of the form or 
model of the objects which could appear in a given 



scene. The only supposition in that the bodies are solid 
objects formed by plane surfaces; in this way, it can 
not find "cubes' 4 - or "houses" in a *oene B since it does 
not know what a "house" ia. Once SEE has partitioned 
a scene into bodie*s P some other program will work on 
them and decide which of those bodies axe "Tiouscs." 

Thus,, SEE ia intended to serve as a link between a 
prc-proceasor M which transforms intensity pietuifls 
into point or line pictures, * and ft recognise? (such *e 
TD 1 or DT'J, which handles this Line picture and 
finds bodies, objects or zcraea matching with certain 
patterns or models. Instead of searching through the 
whole scene looking for parts to match its models, the 
work of the recogniier becomes simpler after SEE has 
partitioned the scene into bodies, because the data to be 
searched (matched) are smaller and better organised. 

The analysis which SEE makes of the different scenes 
generally agrees with human opinion, although m sonic 
ambiguous cases it behaves rather conservatively. Dis- 
tributed over these pages, the reader will find examples 
of scenes analysed by SEE, and the peculiarities and 
behaviorof the program will become clear, 

The program SEE f written in LISP, has been tested 
in the PDP-6 machine of the Artificial Intelligence 
Group, Project MAC, at Massachusetts Institute of 
Technology. A preliminary version, written in CON- 
VERT,* was used extensively for a quick test of ideaa 
which shaped the program to its actual form- The analy- 
sis of a scene takes from 30 to 90 seconds, with the pro- 
gram running interpreted under the interpreter of the 
LISP programming system. 

A more technical description of SEE can be found in 
an unpublished memorandum- T 

Related work 

Rudd H. Canaday 1 in 1962 noalyied scenes com- 
posed of two-dimensional overlapping objects, "straight- 
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FiaUTtE I— TRIAL 

{BODY US* a J) 
(BODY 2 IS ill 112:10) 

(BODYsHSrt r3:5:7>3*:i3i 

• 

■ided piece* of cardboard/' Hift program breaks tlw im*ge 
into ita component part* (the pieces of cardboard) , do- 
scribes each one., tfive* the depth of each part in the 
image (or acene), rod states which parte cover which. 

Roberta p] in 1983 described program* that (1) con- 
vert a picture (a scene) into a line drawing and (2) pro- 
duce a three-diroenaLonal description of the objects 
shown in the drawing in term* of models and their 
tranaformationfl. The main restriction on the lines is 
that they should be * perspective projection of the sur- 
face boundaries of a set of Uiree-diinen»onftl object* 
with planar surfaces- He relies oo perspective and 
numerical computation*, while SEE u&ea a heuristic and 
symbolic (Le., non-numerical ) approach. Also, SEE 
does not need modela to isolate bodies. Roberts' work a 
probably the most important and closest to ours. 

Actually, several research groups (at Massachusetts 
Institute of Technology, l * at Stanford Unmnity, " 
at Stanford Research Institute u ) work actively to- 



wards the realisation of a mechanical manipulator, i.e., 
an intelligent automata who could visually perceive and 
B-i?i>:-33MJy ntcrrt.': with its enviornment, under the 
control of a computer. Naturally , the mechanisation of 
visual perception forma part of their research, and im- 
portant work begins to emerge from them in this area. 

Organization of the paper 

It b formed by the Following headings 

* Introduction and related previous, work- 

* Input Format- The representation of the scene as 
it ift entered to the computer. 

* Format of a Scene. The representation of the scene 
as SEE expects. 

* Type of Vertices. Classification of vertices ac- 
cording to their topology . 

* The program, Analysis of the algorithm, descrip- 
tion of heuristics, 

» Interesting: example*, Diecuaeionr Future work. 

Inputformai 

For testing purposes* the scene* arc catered by hand 
in a simplified format (called input format), and then 
some routines convert this to the form required by SHE . 
Eventually , the data will come from a visual input de- 
vice, through a preprocessor.* 4 

Example* of a ecew 

Suppose w* want to describe the scene 'CUBE. 1 We 
begin by giving (in LISP) a valueto'CUBE-' (Bee Figure 
2 'Cub*') 

(SETQCUBE (QUOTE (A 1,01,001 B AG) 

B 1,0 5j0 (;1 E ;2 C :4 A) 

C 3,07<0(:2Dl4B) 

D 8-0 7-0 (2 E :3 F s4 C) 

E 64 5.0 (£ B -AG -J&D) 

F 8.03.0 03G:4D) 

G 64 1.0 01 A :4 F :3 E) 

))) 

Thus we associate with each vertex its coordinates, 
and a list B in counterclockwise order, of region* and ver- 
tices radiating from that vertex. 

The conversion of the scene, aa just given, into the 
form which SEE expects, fa made by the function 
LLENA; thus, (LLENA CUBE) will put in the prop- 



Decomposition of Usual Scene into Three-DImetisiona! Bodies 293 




Q246 

FIGURE a-'OTBE"— A 



8 




FIGURE 3— 'ONE.' A mho*. V«rtl«e and fiitflaeea dqpaiu) 



erty liat of CUBE the properties REGIONS and 
VERTICES; in the property list of each vertex, the 
properties XCOR, YCOR, NREGIONS, NVERTICES 

and KIND we placed ; and in the property lEst of each 
region, it places the properties NEIGHBORS and 
EVERTICE8. It also marks region :4 as pari of the 

background, 

In other words, LLENA converts a scene from the 
Input FonnaV to the 'Format of a Scene 3 described in 
theoextaftctioD. 



Format of a teem 

A scene is presented to our program as a wene in a 
special symbolic format, which we now describe. 
Essentially, it is an arrangement of relations between 
TBTtiees and regiona. 

A scene has a name which identifies it; this name is an 
atom wbos* property list contains the properties 
-REGIONS/ 'VERTICES 1 and BACKGROUND/ 
For examplej the scene ONE {see Figure 3 'ONE') has 
the name 'ONE/ In the poperty list of 'ONE' we find 

REGIONS — (:I 2 3 :4 $ :6) 

Unordered list of regions composing 
weneONE. 

VERTICES — (ABCDEFGHIJK) 

Unordered list of vertices com- 
posing the scene ONE . 

BACKGROUND— (*) 

Unordered list of regions compos- 
ing the background of the aeeoe 
ONE. 



Region 

A region correspowte to a surface limited by a simple 
connected curve. For instance, in ONE, the surface de- 
limited by the vertices A B C h a region, called ;1, but 
GIJKDHis not- 
Each region hoe as name anatom which poesesseaaddi- 
tbnal properties describing different attributes of the 
region in question. These are 'NEIGHBORS,' 4 KVER- 
TICES,' and *FOOP\ For example, the region in 
scene ONE formed by the line* AE, ED, DK, KC, CA, 
has ':2* at its nam*. In the property li«t of :2 we find; 



NEIGHBORS — 



KVERTICES — 



(3 A ri :l m 

CoanterdockwLse ordered lift of 
all regions which are neighbors to 
:2. For each region, this list is 
unique up to cyclic permutation, 

(DEACK) 

Counterclockwise ordered Hat of 
alt vertices which belong to the 
rsgion :2, This list U unique up to 
cyclic permutation. 



FOOP - (;3D;4EtfA:lC:6K) 

Counterclockwise ordered list of 
alternating neighbors and kver- 
tice* of ;2. This list Ls unique up to 
cyclic permutation. 

Tlie FOOP property of a region is formed by a man 

who walks on tta boundary always having tins region to 
hia left, and takea note of the regions to his right and of 
the vertices which be finds in his way , 
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Vfrtes 

A vertex la the point where two or more Hoes of the 
scene meet; for instance, A, G ? and K axe vertices of the 
leeaeONE* 

Each vertex has as name an atom which posscscB addi- 
tional properties describing different attributes of the 
vertex in question. These are 'XOOR/ 'YCOR,' 
'N VERTICES/ 'NREGIONS/ and 'KIND.* For 
ftxaiupJe, the vertex H (see scene- ONE) has in its prop- 
erty Hrt; 



XCOR 


— 8.0 




x coordinate 


YOOR 


— 15.0 




y -coordinate- 


XVERTICE3 


- (IGD) 




Counterclockwise ordered list of 




vertices to which H is connected. 




This IUt is unique up to cyclic 




permutation. 



NREGIGN8 — <3:S^> 

Counterclockwise ordered list- of 
regions to which H belongs, 
This list is unique up to cyclic 
permutation. 

KIND — (aiiSOslD) 

Counterclockwise ordered list of 

alternating nregions and n vertices 
of H. This list is unique up to eye* 
He permutation. 

Tbe KIND property of a vertex is formed by a man 
who stands at the vertex and, while rotating counter- 
clockwise, take* note of the regions and vertices which 
he Bees. 

NREQIONS and NYERTICES are then easily de- 
rived from KIND: take the odd positioned element* 
of KIND, and its even- positioned elements, respec- 
tively. 

Type* ofttrticti 

Vertices are classified according to the slope, disposi- 
tion and number of lines which form therfl- The func- 
tion (TYPEGENERATOR L), where L is a list of ver- 
tices performs this classification, putting in the prop- 
erty list of each one of the dements of L, tire typ* to 
which U bekttSg*- 

The TYPE of a vertex is always a list of two ele- 
ments; the first is the type-name: one of *L,' r FORK, ? 
-ARROW/ T/ K, s %' 'PEAK/ 'MULTI^ the second 
element is the datum t which generally is a Ust h whew 



form varies with the type-name and contains informa- 
tion in a determined order about the vertex in question. 

{See Table I 'VERTICES'). 

TABLE 1— "VERTICES Ulswiiwilioii of VWtijM td * ttfflA, 
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TA* program 

The program named SEE accepts a scene expressed 
in the notation described above md produces outputs 
IktB identifying and describing the bodies present in the 

ttlflifti 

In this section wo describe the program, and how it 
achieves its goals, by discussing the procedures, 
heuristics etc,, employed and the way they work- 
Wt begin with several examples- 

Example 1. Scene 'STACK* This scene (see Figure 4 
'STACK') is analysed by SEE, with the following re- 
sult*: 

(LOCAL ASSUMES (:&) (41 :W) SAME BODY) 
(BODY 1. IS :1 3 2 :18 :17) 
(BODY 2. IS t4:16:15) 
(BODY3.IS:7A:ll;12) 
(BODY 4. IS 3:8:10) 
(BODY 5. IS 113:14:5) 

(BODY 8.18:20 :1ft) 

Results for scene STACK 

Example 2- Scene *BRIDGE.* With this exwnpte, 
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—■STACK,' This *»• i» analysed by flEE, with 

rweul1« ii*t*iW in F.x*mpl# L. AJl bodiM *ra correctly found. 

Sam* of tfci vartieaa appear in tho drawing with tfamr namee; in. 

other drawing* w« will omit the names; wis in aleo omiMig the 

coordinate e 



we will give details of tbe progra.m'1 operation. We 
start by loading into LISP the pfOCWW for SEE and 
ita auxJi ary functions. 
Then, we evaluate ; 

(UREAD BRIDGE SCEN3) t Q 

Thia causes scene BRIDGE (aw 
Figured r BRIDGE r ),to be wad 

fin the format described pre- 
viously) And tnnwfonftftd to the 
proper form which SEE expects 
(alio deacribed before). 



Finally, we evaluate 

{SEE (QUOTE BRIDGE)) 

This calls SEE to work on 
BRIDGE. 

Results appear in Table II, TtESULTS FOR BRIDGE. 1 



(LOCAL ASSUMES (:IS) (:19) BAME BODY) 
(JiOCAL ASSUMES <2S) <:») SAME BODY) 
(LOCAL ASSUMES (:10) (£ :11 £ rf :4) 9A&IB BODY) 
(LOCAL ASSUMES (.7) (A ill :5 A :4 :10) S AM Ji BODY) 

(BIWOLEBODY ASSUMES (irt :1B> (:H) SAME BODY) 

RESULTS 

(BODY 1. B 0§ :21 £/ US £6) 

(BODY 3, 18 53 36 S3) 
(BODY 3, 18:17 3 £0) 
{BODY 4, 15:1 3) 
flBQDY£.IS:14;l#:l*) 
CBODY«.IB:Mil*j») 
(BODY 7, IS 39 38) 
(BODY*. IS a rtl tf tf :4 :10 J) 



RjMiJU fox trail 'BRUME 1 



TABLE II— RESULTS FOR BRIDGE 

SEE facfc eigfct bodies in aoene 'BRIDGE' (See figure 5 
'BRIDGE*). Pmwnaairy detail turn been renwvwi from (hi 
drawing*; the reader must, bear in mind that Cits coinjdtft* «#ot 
eontaine vertioai (whoee names do not appear Lb (be drawing*) 
and their coordinates (which are also not ahown). *• **H ** #rf ** 1 
and ngfeai. 




FIGURE 5— 'BRIDGE.' The long body 35 34 27 :21 S J3 i 

eonwUj identified. (Bee Table II I 
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SEE and its puts 

The operation of SEE is quite straightforward; the 
program is not recursive and does not do any trvi* 
search. Ita main parte, which are executed one after 
another, unless otherwise indicated,, are: 

FILLSCENE. The properties SLOP and TYPE 
are generated for each vertex, if they were not already 
present. 

CLEAN- Old properties used internally by SEE are 
removed from vertices and regions. 

EVIDENCE. An analysis is made of voices, regions 
andaasociatediiiformation^uiflearchof clucsthat todiflftt* 
that two regions form part of the same body. If evidence 
exists that two regions in fact belong to the same body* 
they are Linked or marked with a "gMurym" (both re- 
cetve the same new label). There are two kinds of Ihika, 
called strong (global) or weak (local), 

LOCALEVIDENCE. Some feature* of the soeae will 
weakly suggest that a group of regions should be oon- 
■ukrod together;, as part- of the same body, This part of 
the program Is that which produce* the local' links or 
evidence. 

GLOBAL. The 'strong* links gathered *> far are ana- 
lysed; regions are grouped into "nuclei" of bodies, 
which grow until some condition* fail to be satisfied (a 
detailed explanation follows later), 

I/X3AL. Weak evidence is taken into account for de- 
riding which of the unsatisfactory global links should be 
considcrtd satisfactory, and the corresponding nuclei of 
bodies arc then joined to form a single and btggtr 
nucleus. Assumption* made by LOCAL are printed 
(see output of BEE), LOCAL may call GLOBAL again, 
or go on. 

SINGLEBODY, If a single region does not belong to 
a larger nucktts, but is linked by one strong evidence to 
another region, it is incorporated into the nucleus of 
that other region. This part of the program may call 
GLOBAL or LOCAL again, if necessary, or continue. 
8JNGLEBQPY also prints its assumptions. 

RESULTS. The regions belonging to the background 
are screened out, and the results are printed. 

Operation 

Here is explained in considerable detail each of the 
parts of SEE that have been sketched above. This will 
help the reader understand the behavior of the program, 
ita strength and deficiencies. 

Example Scene TOWER,' First, we begin by show- 
ing a typical ansly&ia of SEE with a somewhat compli- 
cated scene (see Figure 6 'TOWER'). Meet of the scenes 
•contain, several "nasty" coincidences r a vertex of an ob- 
ject lies precisely on the edge of another object; two 
nearly parallel lines are merged into a single one, etc. 



Thi» has been done on purpose sanoe a non-eo- 
phi*tic*ted preprocessor will tend to make this kind of 
error. 

The output ia given in Table III, TU23ITLT8 
FOR TOWER.' 




FIGURE S— 'TOWER-' Neither LOCAL nor SINGTJvBODY 

an QKem&ry to <sorreotly ptrac this bcsb* Into th# foodies which 

form it. Tlrtf* are reuuiy '■trotag' bilks In tlt» scene and SEE 

ratios good use of ttam. {£*« Table III) 



(BODY 1.18 4 4:1} 

(B0DY3,IS:*:lSa) 
(BODY 3, IS 43:17) 
<BODY4,I8?e:7*) 
(BODY £,JS:W SIS) 
(BODY & 18:13:14:12] 
(BODY7, 18:18 22) 
<B0T>YA.IS4Q:lvt71) 



RbhiIci for TOWER 



TABLE HI— FUwuh* for Figure e 'TOWER' 



F1LISCENE, CLEAN. These two part* of SEE are 
ample ; if necessary, HLLSCENE calb SLOPGEKER- 
ATOR and TYPEGENERATGR; CLEAN remove* 

some unwanted properties, 

EVIDENCE. Strong or global linfca are plaoed by 
this part or the program- Two Junctions are used: 
EVERTICES and TJOINT8- 



Decomposition of Usual Scene into Three-Dimensional Bodies 



EVERTICE3- Thia function considers each vertex of 
the scene under the following rules: 

L. Vertices of the type 'V are not considered, 

FORK. If three regions meet at a vertex of 
the FORK type (Table IV (b» f and 
now of them is the backgrounds links 
betwem them wffij be formed. For instance 
in Figure 6 TOWER/ we establish the 
link* :19— :20, :19- ;2l, and :3&- :21. 
Nevertheless, some of these links may not 
be produced: in Figure 7 'FORK/ the link 
between :3 and :2'm not produced because 
Q«a"pssaingT ? *; the link between :1 and 
#ia regenerated because R is an "VThe 
link between, :1 and ;3 u generated- 

This heuristic is powerful, and (we Figure 
5 'BRIDGE') allows us, for instance, 
while working on vertex jb, to omit tire 
link between regions ;5 and * in scene 
'BRIDGE. 1 Nevertheless, this same heur- 
ristle puts a link between regions & and # 
of the same scene, As we shall see later, 
this situation is not too bad. 

ARROW. This type of vertet (Table IV (c)) cause* 
two of ita regions (the 'left' and the 
'right* one) to be joined; in Table IV (c), 
we will put a lii& between :1 and &, which 
counts as evidence that :1 and £ belong to 
the same body. Nothing is said about & 

For instance, this type of vertex joints 1: 
and 3 in Figure 5 'BRIDGE/ 

X, Two cases art* distinguished: 

{*) TheX is formed by the intersection of 
two line*. No evidence is formed, 

(b) Otherwise (Table IV (f)), links :1— 2 
and £— 4 are formed. This type of X 
occurs when we have piles of objects; 
for instance, in Figure 6 TOWER/ 
rI8 and ;22 are considered i*b belong- 
ing to the same body, due to this type 
of vertex. 

PEAK. All its regions (table IV (h)) F except the one 

containing the obtuse angle, are linked to 
each other, gee also Figure 14 'CORN,' 

T. A search is made for Another T to match the 
vertex currently analysed ; two T's match if 
they are oolinear and "facing each other;" 
if there are several pairs, the closest is 
chosen. For instance (Table IV (d)>, A and 



B are paired, An indicator 'NEXTE* is 

placed in such vertices. 

(a) Once the NEXTE is determined, 
EVERTICE3 establishes ft link be- 
tween ;1 and 2 (Table IV (d)) ( and 
another between =3 and ;4_ These 
links will not be produced if the re- 
sults of them ia to associate the back* 
ground with something that is not 
the background, 
(b) The following test is done (Figure & 
'PARALLEL'): If neither : I or 3 is 
the background, but both have it as 
actKhbor h and in some part of the 
boundary of ;1 (and the same holds 
for ;2) with the background, the seg- 
ment them (M — K in figure 8 
T ARALLEL') is parallel to the cen- 
tral segment (O— P) of the T, then 
;1 and ;2 are linked. For instance, in 
Figure 4 'STACK/ this analysis ap- 
plied to the vertex T will produce 
evidence that ;13 and :14 belong to 
the same body Csince ZA— AA, T— 
MA and OA— NA are parallel). This 
isa rather global heuristic although 
only useful for bodies with parallel 
faees. Also, EVERTICES el&seifias 
T's according to the slope of their 
central segments- 

A summsjy of the strong links put by 
EVERTICES is found to Tabic IV 'GLO- 
BAL EVIDENCE.' 

TJOINTS. This function actuates on T's and es- 
tablished global evidence as described in part (a) of T 
(Table IV (d)). 

LOCALEVTDENCE- An arrow is a leg if one of its 
left or right vertices is a corner (if necessary, through a 
chain of matched T's) which has a side pstfaJkl to the 
central segment of the arrow. The function LEGS finds 
legs in the scene, and stores this information to a list of 
'weak* links (see Figure 9 'LEGS'). 

GLOBAL- Strong evidence is analy red in this part of 
th* program. When this section is entered, several links 
(strong and weak) exist among the different regions of 
the scene. These links are shown pLetorially in Eigure 10 
'LINKS/ for the scene 'BRIDGE' (see both). All the 
links to the background (:30) are deleted: the back- 
ground cannot be part of any body. 

Dtfinitim: a nvkeus (of a body) is either a region or a 
set of nuclei which has been formed by the following 
rule. 
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TABLE IV— 'GLOBAL EVIDENCE' Tbe atnng KnJa are 
represented by doited Una 



> 



u> 




#. 



(CI 



X \ 



VAi 





|D) 



:* 



<IJ 



[»> 




FIGURE 7— TORK/ 

flt*ia; If two nuclei ar* connected by two or more 
links, they are mergeduitoalarger nuekufl by concatena- 

toon- 

(Two nuclei A and B arc tmked if rcgwna a and b are 
linked where a < A and b « B)- For instance, regions :S 
And rll an pot toother, because there exists two links 
among them, to form the nucleus. #— 11- Now, we see 
that region :4 (see figure 10 'LINKS') hw two links 
with thb nucleus --8— 11, and therefore the new nucleus 
58 — 11 »4ia formed- 

We let the nuclei grow and merge under the former 
rule, until no new nuclei can b* formed. When this is the 
cam, tbe scene has been partitioned into several "maxi- 
m*T nuclei; between any two of these there are seto or. 




FIGURE B— 'PARALLELS A link » «rtabUab*d bwtasn :t 
«md 3 because thay do not belong to tfac bwkgtoiiDd, hut the 
h*ekgraund k a neighbor of belli of ihena, and lb* »*m*nt th*t 
Mp*mtea :1 frore thn baegkromwi (and the a»me a true far 2) 
in parallel Ui OP, th* central segjnuut of to* T to quwtioo (:3 it 
the (background)- 








FIGURE 0— 'LEGS/ 
Thrw different t j-jrf* «* le f a - 

at most, one. link. For example, figure 10 'LINKS' will 
be transformed into Figure 11 'NUCLEI/ 

LOCAL. If some strong link joining two "nuurimal" 
nuclei is also reinforced by a weak tusk, these nuclei are 
merged. 

For example, in seed* BRIDGE (see Future 5 
'BRIDGE') tbe following local links exist {among 
others) ; 3 to -4, :10 to :4, :28 to :2ft, :1S to :19- There- 
fore, U*> correspondmg nyclci are merged and now 
fiptrt NUCLEI is transformed into figure 'NEW 
NUCLEI.' . . 

A weak link d*H not cause the regions which it links 
to be merged or eonsklercd as belonging to the same 
body unle&a there is, in addition, one strong evidence be- 
tween such region*. LOCAL may call GLOBAL again, 
if necessary , 
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FIOTBE 10--LLVK9.' TUji fijui* r*flre*eiik. reme 

'BRIDGE/ with the itfong links bctwwin jfci rcf ionu (reprratnted 
here by cirdw) ^Iriiwii. The drilled lin^s. rejw«»nt the evident* 
geneni.i*d by th* vortex PB (m Figure 5 BTUIX3E*!-, The ahort 

arrow ihow Uie links put by vertex JB ; no** tint * link between 

tS and & in e»ot pat. because S (bk Figure & ■'RftTDQB') in * paw- 
ing t-joinL Sg-ziif }|nk» are produced by thn mechanism d«- 
■cribed in part ft) of T. (See Figure S 'PARALLEL') Curled 

bnki are jj«kIuc«1 by vertex GB; evea if this vertex were oc- 
cluded, mid th* links w*m rnnsing, there as itilt enough «vi<fonue 
to identify mfiftnn ;4 5 arid £ rut belonging to the cam* tody. 
Weak Wfilc* art- not shown. 



8INGLEBODY. A strong link joining: a nucleus and 
ft»w>thcr nucleus composed by a single region is con* 
flhfored enough evidence find the nuclei in question 
merged, if there is no other link emanating from the 
tingle region. For instance, in FLjcurv 12 *XEW NU- 
CLEI, ' nucleus ;I8 is merged with nucleus :i& — tlft {se« 
Figure 13 'FINAL'), Nucleus :2S - 29 is not joined with 
:2G— 23— n or with 24—25—27—12—21—9. Even if 
rfic'.i -\in :2ft — 2S were composed by a single region, it 
atUI will not be merged, since two links emerged from it; 
two nuclei elai m it* possession. 

RESULTS. After having screened out the regions 




FIGURE il—'NL'CLCf-' After joinuig all mute hftviaic two 

vt inure link's ill onmniuo, tfcie represenitatinet for the fcefln* 

'BRiOGE' i'h»iiKw from that shown in figure 10 'LJXKS* t*i th* 

uoe shown here. 




FIGURE 12— '%'KW NUCLEI.' Tt.tr figure shows the Srtfi* 

'BRIDGE,' after LOCAL transform* it from the representation 

in Figure 1 1 'NUCLEI' to the one shown here. 



that belong to the background , the nuclei arc prints as 
"todies." 

In this process^ the links which may be joining gome 
of the nuclei are ignored: RESULTS considers- the 
]inkfl of Future 13 'FINAL* ,, for instance, ft» n&n-exifttent. 

Summary 

SEE uses a variety of kinds of evidence to "link" to- 
gether regions of a scene, The links in SEE are supposed 
to be general enough to mftlte SEE an object-analysis 
system. Each link is a piece of evidence that suggest* 
that two or more regions come from the same object* and 
regions that get toed together by enough evidence are 
coiisi'dm'd u* "iv,:i'li;i" >•: [mi: -.'J.- 1 .v.. .■:- 
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(BODY 9. IS ;lO;lfi ill :12) 
{BODY 10.18:18:9:17) 
{BODY 11 IS :7;8) 
{BODY 12. IS 38 37 39) 



FIGURE 13— ■FINAL.' SEXGLGBODY joinU the Lonely r* 
pan :1& witJti th9 ouelw* :lfc-]fl. 

We prraent m this section several scenes witb the- re- 
suite obtained by SEE. 

Example "CORN/ The program analyzes thft toetw 
CORN (hoc figure 14 'CORN') obtaining the following 
identification of bodies : 

{8INGLEBODY ASSUMES (3 3 :1) (;4) SAME 

BODY) 
(BGDYLI8:2i3:li4) 
(BODY 2. IS :7 :6 $) 
(BODY 3. IS :13 ill :12) 

(BODY 4- IS il5 ilS :U) Reauit* for 'CORN 1 

(BODY 5. 18:19:18:17) 
(BODYfl.IS:2L:2Q) 
(BODY7.ISi8:9:lQ) 

The region :4 got a wn«l* link with 3 r aod SINGLE* 
BODY bad w join it with the nueknte :1 =2 ;3. Note 
that A and : 12 did not get joined. The pyramid at the 
top was easily idoatifiod because ita peak produces 
many links. 

Example. *MQM<y, (See Figure IS <MOMO) . The r^ 
suite are aa follows: 

(LOCAL ASSUMES (slT) (£) SAME BODY) 

(LOCAL ASSUMES 1:9 :I7) (:18) SAME BODY} 

(BODY LIS 3 3:1) 

(BODY 2. IS £2 ;33 57 :26] 

(BODY3.ISr28:31) 

(BODY 4, 13 :19 :20 ;34 :30 3&) 

{BODY 5, IS 36:35) 

{BODY*}. IS 3*4 £31 A) 

(BODY 7. IS :25 3$ :22) RetulU for *MOMO J 

(BOL>Y8.ISrI4:13;lS) 




FIGURE U— 'CORN.* SIk* a link betrani :4 aad :12 ■ not 

ftriabbabed, the hotfi** foufid ill thai part cf the Meoe are 

:l:^4ud:l) 213:13. 




FIGURE 15— 'MOMO/ 
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Comments on the solution for 'MQMQ*: The central 
cubes wft easily isolated. 31 gets & link with :5 and 
another vrlth :24, so there ia no need to msfc LOCAL La it. 
Til* same is true of :26 with :27 and #3. 

There is enough strong evidence to joint ;28 and £1. 

Thy links for -29 :30 34 30 :19 md # :17 ;19 an in- 
dicated in Figure IG •MO.MO-LINKS.' 

The dot tod Links between regions 3Q and :34 9 And be- 
tween :20 acid :10 (so* Figure 15 ' MOMO') at* du# lathe 
heuristic of parallel lines {of Figure 8 'PARALLEL'), 
These links made it possible to join the otherwise dis- 
conjiected nuclei ;29- ;30- :19and :34- :20. In particular, 
If $4 or SO did not have parallel sidea, SEE would fowt 
fatted to merge these nuclei, and would have reported 
two bodies The disposition, of regions in MOMQ is such 
thftt no link is present between :29 and :34, since :2S 
*nd ;$1 fall "exactly" parallel to :2» and -TO. In a less 
extreme ftfceiw, some of these links would be present, al- 
loming cwtect identification even if 2&-:3GV34-20-:19 
were not * prism. Anyway, SEE did not make any mis- 
tahei- 

The triangle of links formed by 29, :1S and :17 nor- 
mally would huve produced 3 separate bodies (sanee we 
need at Lea&t one double link to merge two regions) ; 
nevertheless, in this case, LOCAL makes some assump- 
tions and saves the situation. Again, if :9 were not a 
parallelogram , there would be no weak Jinks between :9 
and ;18 or between :9 and ;17 r and 3 bodies would have 
been reported instead of :9-:17-:lS. But we should no* 
tice that, in general, two links instead of one would be 
between ;1 7 and ;1S. 

Link* were established between ;12 and ;13, without 
*eriou» consequences, because we need two mistakes, 
two wrong atrong links, to fool the program. But that 
could happen. 



Example. 'HOME*,— (see scene- HOME). The results 
are in Tabte V 'RESULTS FOR HOM & 1 




(SIXGLEBOnY ASSUMES (:3ft) (3ft> SAME BODY) 

(SINCLEBQDY ASSUMES (riJ4) (33) SAME BODY) 

(SlNGLEBODY ASSUMES (:2S :24 33) (3ft SAME BODY! 

«1NULEB0DY ASSUME -" ?-T. :i\ >\m;: ■;,. .1. V 

RESULTS 

(BODYl.TS:a:fi:10:2> 

(BODY2.TS:l4:ta:U :I2) 

(BODY a. IS 3 A 

fBODY^ISilSl 

(BODYJk IS :ia :fV> 

(BODY ft- 13 :3ft :3ft} 

(BODYT.IS :7 36:271 

(BODY S. 13:20 £11 

(BODY fl. 18:29:28) 

(BODY 10. IS M :33) 

CBODV II. IS £i :32 :30} 

(BODY 12. IS 125 :2-1 :22 23) 

(BODY 13. IS :10> 

(BODY H. IS ilB) 

>IBODYl5,T8:17> 

(BODY 16, IS :36 37^5^ 

TABLE V— Remits far HOME 



Results for HOME 




FIGURE lft— MOMO— LINKS'. Sere* 
fipjre 15 'MOMO* 



Unto* of 



FIGURE IT— 'HOME.* The hexnfloAal prterm did oo* «ntl« 
dinVultira; SlNGLEBODY wu Needed to Rruup ^4 with ^3. 
The body :3S-39 ins kteatiiieii m « ttibcle one, but J6-17 w« 
reported, u two. Note tbM. tbtr* tftott no* ««Ut Iwd kink between 
21 *nd 20; nevertheteea, SlNGLEBODY makes the correct 
Iklentlfltttkm- (Sm Table V). 



Cbrnttumfft on the solution to HOME: There ia a certain 
degree of ambiguity in this arane which hunwi beings 
tend to resolve by awummK pnndk!i?pLpedfl- :16 and :17 
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arc reported as two separate bodies, instead of one, 
which is probably a more natural answer. 

In th* picture from which 'HOME' was drawn, 
:19 and ;18 were the two faces of a single parallelepiped 
leaning on :38-;39; SEE reports ; 19 as one body and ;18 
H4 Another. 

Nevertheless, SEE report* 38 and r39 as fomtoe 
part of the same body (which was in fact the ease in the 
picture in question), due to the fact that ;4 and :1 are 

background. 
Example- SPREAD.'— (*ee Figure IS 'SPREAD*). 

The mutta ar* in Table VI 'RESULTS FOR SPREAD'. 



(LOCAL A5SITMES f:36> (*) 8AMRBODY"! 

{LOCAL ASSUMES (40) (31 42 :2ft) SAME BODY 

(LOCAL ASSUMES (:J() (:|7) SJAME BODY) 

(LOCAL ASSUMES (:S) C=3Q) SAME BODY) 

(LOCAL ASSUMES (3 :li rfl) (:12 :]01 SAME BODY) 

(B1NGLEBODY ASSUMES f:23'| (22) SAME B0DY1 

(BINGLKBODY .ASSUMES {A M\ (37) SAME BODY) 

(SlNGLEBODY ASSUMES (3S :36 SHl (36) SAME BODY) 

RESULTS 

(BOI>¥ 1, IS 39 9*0 136) 

(BODY 2. IS £3:22) 

(BODY 3.18^:41) 

(BODY4. 18*30 Si 1 } 

(BODY MS rS*> 

(BODY 6. IS :I» :& :37 &5J 

(BODY7.IB31iS2^i30) 

(BODY S, IS :30 a) 

(BODYft.I8:i2U0 4:11 * 

(BODY 10. IB :t3 :7 :I) 

(BODY 13, IS 21 :B) 

(BODY 12. IS :3 ||fl 

(BODY 13. IS :17 :Ifl) 

(BODY 14. IS *5 :43 :44) 

(BODYia.lS:lB> 

(BODY 16. IS :15 3l4j 



RMUlte *«>r SPREAD 



TABLt; VJ— K^-jI'* n... SPMFAU 



Cffhunaris <m ihe tottfum to SPREAD: The body 
;22*23~24 a due to insuinefceacy of links, was aplit In two : 
22-23 and $4, 

Since there is only one link between & and :5, tbia 
body gets spite into two: :G-21 and :5-20. Note that :21 
ift noC the aame face as :2Q, and there is where SEE gets 
confused and refuses to eoe evidence toward Linking ;21 
with :20. 

Th* long body ;9~lO-l 1-12^3 gets properly identified. 
Example. 'HARD.*— Thift *oe*ft U analysed with the 
following result*: 

(LOCA I, ASSUMES ( :1 1 ) ( :12) SAME BODY) 
(LOCAL ASSUMES (:15) (:I6) SAME BODY) 
RESULTS 
(BODY 1. IS ;13 rll) 



{BODY2.I8;l6rlo) 

(BODY 3.18 ;32 31:30) 

(BODY 4, IS 0:10 *) 

(BODY5,IS:18:10;20) 

(BODY »- 13:13:17:14) 

(BODY7-IS:5:4) 

(BODY 8. 18:1 2*3) 

(BODY 9- IS £4 =23 32 :3 31 :28 :2Q) 

(BODY 10- IS 3S flft 37) 

(BODY 11. IS :7) 

(BODY12_TS:6) 



ReMult*i f<Kr f HARD' 




FIGURE 1A— "SPREAD. • .41 and :42 are kfoaufled u a 

fltnfle body, Nev*rtb&h*s, :6-18-19 R««to broken into #49 aod 

lift, r2s\-37-9fr-2A geta enrhwtly uk«itifl*d t M w*U u the funny 

iMking :29~30-3t-32. (Stt»T*hk VI}. 



Cowmen** on Af «et[u(Mm to HARD: r!5 Mid :16 have 

to make use of iveak. evidence to get joined and recog- 
nized as forming part of the same body- Nevertheless, 
toi* 38 not necessary with 28 and :29, because, through 
& chain of T&, there ia enough evidence in :3, :21 and ;22 
toiomaucceasfullyalithaUonRand twi^e occluded body. 
There La one serious bufl in this identification: regions 
:7 and :6 get identified as two separate bodies, aad not 
as a aingjc one, as one would normally do. Thin ka caused 
by the fact that neither :7 nor :6 have visible 'useful' ver- 
tices* And there are not enough parallel Itnea in them to 
use tie heuristic of Figure H 'PARALLEL/ 
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FIGURE 19— 'HARD.' Badv :2L-22-3-23-2i-2S-29 in reported 

u * BiD*lr object, which a correct- Nevertheless, re^iaoa :7 And :6 

get reported u two bodies. 



:33 was recognised as part of ;l-2. as it should be, 

DISCUSSION 

We have described a program that analyzes a three-di- 
mensional scene (presented in the form of a Line draw- 
jag) And splits it into "objects" on the basis of pure 
form. If we consider a scene as a set of regions (sur- 
faces) , then SEE partitions the set into appropriate sub- 
sets, each subset forming a three-dimensional body or 
object. 

The- performance of SEE shows to us that U is possible 
to separate a scene into the objects forming it , without mtd~ 
inp to know in deUtil these objects; SEE does not need 
to know the 'definitions 1, or descriptions of a pyramid , or 
a pentagonal prism, in order to isolate these objects in a 
scene containing 1Lim:: : . ^vi'n in :hf ca-n; '.vl>rn i'ney an- 
partially occluded. 

The basic idea behind BEE is to make global use of in- 
formation collected loyally at each vertex: this informa- 
tion is noisy and SEE has ways to combine many dif- 
ferent kinds of unreliable evidence to make fairly re- 
liable global judgmental. 

The essentials are: 

(1) Representation as vertices (with coordinate*), 

line? and regions 
(2} Types of vertices, 
(3) Concepts of links (strong and weak), nuclei and 

rules for forming them . 

The current version of SEE is restricted to scenes) pre- 



sented in symbolic form. It does not have resources for 
dealing with curves, shadows, noise, missing lines, etc. 
So it represents a "geometric theory of object identity" 
at present. 

Since SEE requires two strong evidences to join two 
nuclei, it appears that its judgment* will Lie in the 
'safe* side, that is, SEE will almost never join two re- 
gions that belong to different bodies. From th? analysis 
of scenes shown above, its errors are almost always of 
the same type: regions that should be joined are left 
separated- We could say that SEE behave*" conserve 
atively," especially in the presence of ambiguities. 
Division* of the evidence into two types, strong and 
weak, results In a good compromise, The weak evidence 
is considered to favor linking the regions, but this evi- 
dene* is used only to reinforce evidence from more re- 
liable clues. Indeed, the weak links that give extra 
weight to nearly parallel lines arc a concession to ob- 
ject*recognttion a in the sense of letting the analysis sys- 
tem exploit the fact that rectangular objects are com- 
mon enough in the real world to warrant special atten- 
tion. 

Most of the ideas in SEE will work on curves too. 
H.:>v.t-, l -r, wg il'.: not y..; have n good idea of how ^ctjf:- 
tive the system will be to "symbolic noise," i.e. , missing 
misplaced, and False region boundaries. As indicated in 
the scenes above, the system seems to have good re- 
sistance to "acoidenta" in which object* happen to 
to 4 itae up" unusually well. This feature may be neces- 
sary if the preprocessor that (eventually) will feed data 
SEE decides to report two nearly colinear lines as one 
alone, or if it lumps several vertices into one, because 
they lie close to each other. 

Extensions. (None of this incorporated in the actual 
program.) More heuristics could be added to increase 
the number of links; in particular, given a good number 
of "link proposers," parameters: set outside (by the 
user) would tell SEE which set of heuristics to use; for 
instance, if we knew that the scene is formed by prisma, 
we could uae the heuristics that ask for parallel lines 
having ft given configuration, we could check the length 
of certain edges, etc. 

DiBerent kinds of links could also be established; in 
this way, 'contradictory 1 links (such as the three links of 
Figure 13 'FINAL' which SEE just ignores) could be 
studied further in order to discover ambiguities. In 
particular, a "conditional link" would be useful r regions 
2 and ;3 belong to the same body if region ;4 does not. 
S1NGLEBODY could be improved so as to analyze 
in more detail the regions that by themselves form 
bodies (that is, the bodies formed by only one region}; 
in this way, we could hope to joint regions :6 and :7 of 
scene 'HARD.' 
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ADDENDUM 

The work reported here has been extended in several ways: 

handling of stereo information {two views, left and right, of 

the same scene) , improvements to deal with noisy (imperfect) 

input, f igure-bacXground discrimination, optical illusions and 

some aspects of psychological perception. The interested 

reader could find these in; 

Guzman, A, Computer Recognition of Three- Dimensional 
objects in a visual scene, Ph.D. Thesis, Electrical 
Engineering Department, M.I.T., December I960. Avail- 
able also as a Project MAC Technical Report MAOTR-59* 
AD number AD 692 200* 

This thesis also describes an improved version of SEE,, and gives 

more exanples of scenes decomposed by SEE- 

Of course, after the bodies have been found in a scene, it 

is natural to ask what they are, to classify them: Is body 

(:3 :7 :11) a pyramid? To answer this, we introduce the concept 

of model as a representation or description of classes of objects: 

"pyramid", "house", "triangular prism", etc* TD and DT are two 

programs that search a visual scene for instances of a given 

model (for example r find all the cubes present in such scene) . 

TD is described in? 



Guzman, A. Scene Analysis Using the Concept of Model- 
Report AFCRL-670133* Computer Corporation of America, 
565 Technology Square, Cambridge, Mass. January 1967. 
AD-6 52-017. 

DT is described in: 

Guzman, A. A Primitive Recognizer of Figures in a Scene* 
Project MAC Memorandum MAC-M-342 (A* I* Vision Memo 119) . 
M.I.T. January 1967* 

A general discussion (and an attempt At classification) of 

models,, their properties and ways to identify objects (poly- 

hedra) of known forms is found in- 

Guzman, A* Some Aspects of Pattern Recognition by Computer* 
M,S* Thesis f Electrical Engineering Department, M*I*T» r 
February 1967* AD-656-041* Also available as a Project 
MAC Technical Report MAC TR 37, 

DT and TD work with models and scenes made of straight lines? 

they do not handle correctly objects partially occluded. 

Both TD and DT could search for any model (which can be 
described or constructed in certain figure description language) 
in a scene. Another approach to object recognition is to have 
specific programs to look for a specific model (or a restricted 
class of models) . In this spirit, POLYBRICK is a program that 
searches for parallelepipeds in a scene described as a line 
drawing; occluded cases are correctly handled. 

The references ares 

Guzman , A. POLYBRICK j Adventures in the Domain of 



Parallelepipeds. Project MAC Memorandum MAC-M-308 
{A* I. Memo 96), M.I.T, May 1966. (Also found in 
KAC-TR-37.) 

Guzman, A. Object Recognition: Discovering the Parallele- 
pipeds in a Visual Scene. Proceedings of the Second Hawaii 
International Conference on Systems Sciences, University of 
Hawaii, Honolulu, Hawaii, January 1969- Western Periodicals 
Co. pp. 479-82. 



NOTES: Reports with AD numbers (such as AD-652-017) can also be 
obtained as follows: 

Government contractors iray obtain copies of the report frora 
the Defense Documentation Center, Document Service Center, 
Cameron Station, Alexandria, Va • 22314, Orders will be expedited 
froro DDC if placed by your librarian, or some other person author- 
ized to request documents. 

Other U,S, citizens and organisations may obtain copies of 
this report froiti the clearinghouse for Federal Scientific and 
Technical Information (CFSTI) , Sills Building, 5285 Port Royal Rd., 
Springfield, Va . 22151. Cost: $3.00 each. 



